Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizers

Automated hyperparameter optimization (HPO) has gained great popularity and is an important component of most automated machine learning frameworks. However, the process designing HPO algorithms still unsystematic manual process: new are often built on top prior work, where limitations identified improvements proposed. Even though this approach guided by expert knowledge, it somewhat arbitrary. The rarely allows for gaining a holistic understanding which algorithmic components drive performance carries risk overlooking good design choices. We present principled to benchmark-driven algorithm applied multifidelity (MF-HPO). First, we formalize rich space MF-HPO candidates that includes, but not limited to, common existing then configurable framework covering space. To find best candidate automatically systematically, follow programming-by-optimization search over via Bayesian optimization. challenge whether found choices necessary or could be replaced more naive simpler ones performing ablation analysis. observe using relatively simple configuration (in some ways, than established methods) performs very well as long critical parameters set right value.